{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Pr\u00e0ctiques de Nous Usos de la Inform\u00e0tica"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from time import clock as time\n",
      "import pandas as pd\n",
      "import numpy as np\n",
      "import math\n",
      "\n",
      "base_url = '/media/server/ftp/TNUI/practica2/ml-1m/'\n",
      "#base_url = '/home/jmarinve7.alumnes/Share/TNUI/practica2/ml-1m/'\n",
      "#base_url = '/home/maikel/'\n",
      "unames = ['user_id', 'gender', 'age', 'occupation', 'zip']\n",
      "users = pd.read_table(base_url + 'users.dat', sep='::', header=None, names=unames)\n",
      "rnames = ['user_id', 'movie_id', 'rating', 'timestamp']\n",
      "ratings = pd.read_table(base_url + 'ratings.dat', sep='::', header=None, names=rnames)\n",
      "mnames = ['movie_id', 'title', 'genres']\n",
      "movies = pd.read_table(base_url + 'movies.dat', sep='::', header=None, names=mnames)\n",
      "\n",
      "data = pd.merge(pd.merge(ratings, users), movies)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 23
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<b> EXERCICI 1:</b>\n",
      "Calcula la puntuaci\u00f3 mitjana de cada usuari. Quina \u00e9s la pel\u00b7l\u00edcula m\u00e9s ben puntuada? "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# mean_rating_by_user ha de ser un dataframe que cont\u00e9 el nom del usuari i la mitja de les seves puntiacions\n",
      "mean_rating_by_user = pd.DataFrame(data.pivot_table('rating',  cols='user_id', aggfunc='mean'))\n",
      "# critic_higher_mean ha de ser el registre de l'usuari amb la mitja m\u00e9s elevada\n",
      "critic_higher_mean = mean_rating_by_user.sort_index(by='rating', ascending=False)[:1]\n",
      "# hacemos la media de las movies y guardamos su titulo, esta vez lo hacemos sin dataframe\n",
      "mean_movies = data.pivot_table('rating', cols='title', aggfunc='mean')\n",
      "# sacamos la pelicula con la media mas alta media mas alta, comola media mas alta es 5\n",
      "# sacamos todas las peliculas con media de 5\n",
      "max_rating_movies = mean_movies[mean_movies.values == mean_movies.values.max()]\n",
      "print max_rating_movies"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "title\n",
        "Baby, The (1973)                             5\n",
        "Bittersweet Motel (2000)                     5\n",
        "Follow the Bitch (1998)                      5\n",
        "Gate of Heavenly Peace, The (1995)           5\n",
        "Lured (1947)                                 5\n",
        "One Little Indian (1973)                     5\n",
        "Schlafes Bruder (Brother of Sleep) (1995)    5\n",
        "Smashing Time (1967)                         5\n",
        "Song of Freedom (1936)                       5\n",
        "Ulysses (Ulisse) (1954)                      5\n",
        "Name: rating, dtype: float64\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<b>EXERCICI 2:</b> Defineix una funci\u00f3 anomenada <b>top_movie</b> que donat un usuari ens retorni quina \u00e9s la pel\u00b7l\u00edcula millor puntuada.<br> \n",
      "def top_movie(user)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def top_movie(dataFrame,usr):\n",
      "    \"\"\"\n",
      "    Retorna la primera pelicula amb el valor mes alt puntuat per el usr\n",
      "    \"\"\"\n",
      "    # el data del usr\n",
      "    user_dataframe = dataFrame[dataFrame.user_id == usr]\n",
      "    # la primera pelicula de les que tenen la puntuacio mes alta del usr\n",
      "    return  user_dataframe[user_dataframe.rating == user_dataframe['rating'].max()][:1]['title']\n",
      "    \n",
      "\n",
      "print top_movie(data,4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "45686    Run Lola Run (Lola rennt) (1998)\n",
        "Name: title, dtype: object\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<b>Exercici 3: </b>\n",
      "\n",
      "Construeix dues funcions, <b>distEuclid(x,y)</b> i <b>coefPearson(x,y)</b>, que implementin la dist\u00e0ncia Euclidiana i el coeficient de correlaci\u00f3 de Pearson entre dos vectors. Escriu les funcions que calculin la semblan\u00e7a entre dos usuaris segons aquesta estructura:\n",
      "\n",
      "<b>def SimEuclid (DataFrame, User1, User2)</b>\n",
      "    Calcular els vectors representatius de cada usuari, C1 i C2, amb les puntuacions dels \u00edtems comuns que han puntuat el dos usuaris.<br>\n",
      "    Si no hi ha puntuacions en com\u00fa, retornar 0.\n",
      "    Retornar 1/(1+distEuclid(C1, C2))<br>\n",
      "    \n",
      "<b>def SimPearson (DataFrame, User1, User2)</b>\n",
      "    Calcular els vectors representatius de cada usuari, C1 i C2, amb les puntuacions dels \u00edtems comuns que han puntuat el dos usuaris.<br>\n",
      "    Si no hi ha puntuacions en com\u00fa, retornar 0.>\n",
      "    Retornar coefPearson(C1,C2)<br>\n",
      "    \n",
      "\n",
      "<b>Utilizeu el panda per a realitzaci\u00f3 d'aquest exercici.</b>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Returns the euclidean distance of to vectors\n",
      "def distEuclid(x, y):\n",
      "    \"\"\"Funcio que retorna la distancia euclidiana entre dos arrays\"\"\"\n",
      "    return ( ( x - y ) ** 2 ).sum() ** 0.5\n",
      "\n",
      "\n",
      "\n",
      " # Returns the euclidean pearson of to vectors \n",
      "def coefPearson(x, y):\n",
      "    \"\"\"Funcio que retorna el coeficient de pearson per a dos arrays\"\"\"\n",
      "    #: mitjana de l'user x\n",
      "    x_mean = x.mean()\n",
      "    #: mitjana de l'user y\n",
      "    y_mean = y.mean()\n",
      "    #: denominador de l'user x\n",
      "    den_x = math.sqrt(( (x - x_mean) ** 2 ).sum())\n",
      "    #: denominador de l'user y\n",
      "    den_y = math.sqrt(( (y - y_mean) ** 2 ).sum())\n",
      "    #: denominador\n",
      "    den = den_x * den_y \n",
      "    if den == 0:\n",
      "        return 0\n",
      "    \n",
      "    #: numerador del coeficient de pearson\n",
      "    num = ( (x - x_mean) * ( y - y_mean ) ).sum()\n",
      "    # retornem l'operacio\n",
      "    return num / den\n",
      "\n",
      "\n",
      "\n",
      "# Returns a distance-based similarity score for person1 and person2 based on euclidean distance\n",
      "def SimEuclid(DataFrame,User1,User2):\n",
      "    users_frame =  pd.merge(\n",
      "                        DataFrame[DataFrame.user_id == User1][['movie_id','rating']], \n",
      "                        DataFrame[DataFrame.user_id ==User2][['movie_id','rating']],\n",
      "                        on='movie_id', suffixes=('-'+str(User1),'-'+str(User2))\n",
      "                    )\n",
      "    \n",
      "\n",
      "    return distEuclid(users_frame['rating-'+str(User1)].values,users_frame['rating-'+str(User2)].values)\n",
      "\n",
      "\n",
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimPearson(DataFrame,User1,User2):\n",
      "    #: merge dels dos usuaris\n",
      "    users_frame =  pd.merge(\n",
      "                    DataFrame[DataFrame.user_id == User1][['movie_id','rating']], \n",
      "                    DataFrame[DataFrame.user_id ==User2][['movie_id','rating']],\n",
      "                    on='movie_id', suffixes=('-'+str(User1),'-'+str(User2)))\n",
      "    \n",
      "    return coefPearson(users_frame['rating-' + str(User1) ].values,\n",
      "                       users_frame['rating-'+str(User2)].values)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Execute functions\n",
      "init = time()\n",
      "print SimEuclid(data,1,3)\n",
      "end = time()\n",
      "print \"tiempo: \", (end-init), \"s\"\n",
      "init = time()\n",
      "print SimPearson(data,1,3)\n",
      "end = time()\n",
      "print \"tiempo: \", (end-init), \"s\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3.31662479036\n",
        "tiempo:  0.02 s\n",
        "-0.332181919415\n",
        "tiempo:  0.01 s\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<b>Exercici 4:</b>\n",
      "\n",
      "Feu dues funcions, <b>getNBestEuclid(DataFrame, user,n)</b> i <b>getNBestPearson(DataFrame, user,n)</b>, que retornin els n usuaris m\u00e9s semblants segons aquestes dues mesures de similitud."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimPearson(DataFrame,User2):\n",
      "    main_frame = DataFrame[DataFrame.user_id == User2]\n",
      "    return coefPearson(main_frame['base_user_rating'].values, main_frame['rating'].values)\n",
      "\n",
      "\n",
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimEuclid(DataFrame,User2):\n",
      "    main_frame = DataFrame[DataFrame.user_id == User2]\n",
      "    return 1 / (1 + distEuclid(main_frame['base_user_rating'].values, main_frame['rating'].values))\n",
      "\n",
      "# return the N most similar users to given used based on euclidean distance\n",
      "def getNBestEuclid(DataFrame,user,n):\n",
      "        # usuario base\n",
      "    base_user_id = user\n",
      "    # generamos dataframe con las movie_id del base user y sus ratings\n",
      "    base_user_frame = DataFrame[DataFrame.user_id == base_user_id][['movie_id','rating']]\n",
      "    # renombramos las columnas\n",
      "    base_user_frame.columns = ['movie_id', 'base_user_rating']\n",
      "    # generamos dataframe con los registros que esta el base user y solo las columnas movie_id user_id rating\n",
      "    base_no_user_frame = DataFrame[ DataFrame.user_id != base_user_id ][['movie_id', 'user_id', 'rating']]\n",
      "    # hacemos un dataframe a partir de un merge, donde mantenemos todas las movies del base_user (que son las que\n",
      "    # tienen en comun ambos dataframes), el rating del base user , y las id de los otros users con sus ratings\n",
      "    # : Por cada movie_id y rating del base user, estaran todos los demas\n",
      "    # : y tendremso los datos necesarios en el mismo registro\n",
      "    base_frame = pd.merge(base_user_frame,base_no_user_frame, on='movie_id')\n",
      "    # id's unicas de usuarios que no son base user\n",
      "    users_id = base_no_user_frame['user_id'].unique()\n",
      "    \n",
      "    #: dataframe de sortida\n",
      "    result_frame = pd.DataFrame()\n",
      "    #: distancias que acumularemos\n",
      "    distancies = [ SimEuclid(base_frame,user_id) for user_id in users_id ]\n",
      "        \n",
      "    # juntamos todos los calculos en un dataframe\n",
      "    results = pd.DataFrame(data = {'user_id':  users_id, 'similarity':distancies })\n",
      "    return results.sort(['similarity'],ascending=False)[:n]['user_id'].values\n",
      "    # devolvemos el user y la similarity\n",
      "    #return results.sort(['similarity'],ascending=False)[:n]\n",
      "\n",
      "# return the N most similar users to given used based on pearson distance\n",
      "def getNBestPearson(DataFrame,user,n):\n",
      "    # usuario base\n",
      "    base_user_id = user\n",
      "    # generamos dataframe con las movie_id del base user y sus ratings\n",
      "    base_user_frame = DataFrame[DataFrame.user_id == base_user_id][['movie_id','rating']]\n",
      "    # renombramos las columnas\n",
      "    base_user_frame.columns = ['movie_id', 'base_user_rating']\n",
      "    # generamos dataframe con los registros que esta el base user y solo las columnas movie_id user_id rating\n",
      "    base_no_user_frame = DataFrame[ DataFrame.user_id != base_user_id ][['movie_id', 'user_id', 'rating']]\n",
      "    # hacemos un dataframe a partir de un merge, donde mantenemos todas las movies del base_user (que son las que\n",
      "    # tienen en comun ambos dataframes), el rating del base user , y las id de los otros users con sus ratings\n",
      "    # : Por cada movie_id y rating del base user, estaran todos los demas\n",
      "    # : y tendremso los datos necesarios en el mismo registro\n",
      "    base_frame = pd.merge(base_user_frame,base_no_user_frame, on='movie_id')\n",
      "    # id's unicas de usuarios que no son base user\n",
      "    users_id = base_no_user_frame['user_id'].unique()\n",
      "    \n",
      "    #: dataframe de sortida\n",
      "    result_frame = pd.DataFrame()\n",
      "    #: distancias que acumularemos\n",
      "    distancies = [ SimPearson(base_frame,user_id) for user_id in users_id ]\n",
      "        \n",
      "    # juntamos todos los calculos en un dataframe\n",
      "    results = pd.DataFrame(data = {'user_id':  users_id, 'similarity':distancies })\n",
      "    return results.sort(['similarity'],ascending=False)[:n]['user_id'].values\n",
      "    # devolvemos el user y la similarity\n",
      "    #return results.sort(['similarity'],ascending=False)[:n]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# EXECUTE FUNCTIONS\n",
      "init = time()\n",
      "print getNBestEuclid(data,1,5)\n",
      "print \"tiempo: \",time()-init\n",
      "init = time()\n",
      "print getNBestPearson(data,1,5)\n",
      "print \"tiempo: \",time()-init\n",
      "#[4211 4664 4045 3818   61]\n",
      "#[3725 3635  565  448 4106]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[4211 4664 4045 3818   61]\n",
        "tiempo:  3.53\n",
        "[3725 3635  565  448 4106]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "tiempo:  3.84\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<b>Exercici 5:</b>\n",
      "\n",
      "Desenvolupa un sistema de recomanaci\u00f3 col\u00b7laboratiu basat en usuaris. La funci\u00f3 principal, <b>getRecommendationsUser</b>, ha de tenir com a entrada una taula de puntuacions, un \"user_id\", el tipus de mesura de semblan\u00e7a (Euclidiana o Pearson) que volem usar i el nombre n m\u00e0xim de recomanacions que volem. Com a sortida ha de donar la llista de les 5 millors pel\u00b7l\u00edcules que li podriem recomanar segons la seva semblan\u00e7a amb altres usuaris.\n",
      "\n",
      "Nota: S'ha d'evitar comparar \"user_id\" a ell mateix."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimPearson(DataFrame,User2):\n",
      "    main_frame = DataFrame[DataFrame.user_id == User2]\n",
      "    return coefPearson(main_frame['base_user_rating'].values, main_frame['rating'].values)\n",
      "\n",
      "\n",
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimEuclid(DataFrame,User2):\n",
      "    main_frame = DataFrame[DataFrame.user_id == User2]\n",
      "    return 1 / ( 1 + distEuclid(main_frame['base_user_rating'].values, main_frame['rating'].values) )\n",
      "\n",
      "\n",
      "\n",
      "# Gets recommendations for a person by using a weighted average\n",
      "# of every other user's rankings\n",
      "\n",
      "def getRecommendationsUser(DataFrame,person,n,similarity=SimPearson):\n",
      "    # usuario base\n",
      "    base_user_id = person\n",
      "    # generamos dataframe con las movie_id del base user y sus ratings\n",
      "    base_user_frame = DataFrame[DataFrame.user_id == base_user_id][['movie_id','rating']]\n",
      "    # renombramos las columnas\n",
      "    base_user_frame.columns = ['movie_id', 'base_user_rating']\n",
      "    # generamos dataframe con los registros que esta el base user y solo las columnas movie_id user_id rating\n",
      "    base_no_user_frame = DataFrame[ DataFrame.user_id != base_user_id ][['movie_id', 'user_id', 'rating']]\n",
      "    # hacemos un dataframe a partir de un merge, donde mantenemos todas las movies del base_user (que son las que\n",
      "    # tienen en comun ambos dataframes), el rating del base user , y las id de los otros users con sus ratings\n",
      "    # : Por cada movie_id y rating del base user, estaran todos los demas\n",
      "    # : y tendremso los datos necesarios en el mismo registro\n",
      "    base_frame = pd.merge(base_user_frame,base_no_user_frame, on='movie_id')\n",
      "    # id's unicas de usuarios que no son base user\n",
      "    users_id = base_no_user_frame['user_id'].unique()\n",
      "\n",
      "    \n",
      "    #: dataframe de sortida\n",
      "    result_frame = pd.DataFrame()\n",
      "    #: distancias que acumularemos\n",
      "    distancies = [ similarity(base_frame,user_id) for user_id in users_id ]\n",
      "    # juntamos todos los calculos en un dataframe\n",
      "    similarities = pd.DataFrame(data = {'user_id':  users_id, 'similarity':distancies })\n",
      "    \n",
      "    # nos quedamos solo con los similarities mayores que 0\n",
      "    similarities = similarities[similarities.similarity > 0]\n",
      "    \n",
      "    # listado de peliculas unicas que hay en el dataframe\n",
      "    unique_movies = DataFrame['movie_id'].unique()\n",
      "    # creamos un listado de peliculas, unicas, y que no ha visto el user base\n",
      "    no_user_movies = [ x for x in unique_movies if x not in base_user_frame['movie_id'].unique() ]\n",
      "\n",
      "    # creamos un data_frame con las no_user_movies\n",
      "    recomendations = pd.DataFrame(data = no_user_movies, columns = ['movie_id'])\n",
      "    # hacemos un merge dataframe con : movie_id, user_id, rating : y no_user_movies : on movie_id\n",
      "    recomendations = pd.merge(recomendations, DataFrame[['user_id','movie_id','rating',]], on='movie_id')\n",
      "    # lo unimos con los similarities : on user_id\n",
      "    recomendations = pd.merge(recomendations,similarities, on='user_id')\n",
      "    \n",
      "    sim_sum_total = []\n",
      "    no_user_movies = recomendations.movie_id.unique()\n",
      "    \n",
      "    #recorremos las movies no vistas que se pueden recomendar\n",
      "    for movie in no_user_movies:\n",
      "        # extraemos los ratings y los similaritys de los user que la han valorado\n",
      "        step_data = recomendations[recomendations.movie_id == movie][['rating','similarity']]\n",
      "        # total rating \n",
      "        total = ( step_data.rating * step_data.similarity ).sum()\n",
      "        # total similarity\n",
      "        sim_sum = step_data.similarity.sum()\n",
      "        # prediccion\n",
      "        sim_sum_total.append( total / sim_sum )\n",
      "        \n",
      "    #lo metemos todo en un dataframe\n",
      "    predictions = pd.DataFrame(data = {'movie_id':  no_user_movies, 'prediction':sim_sum_total })\n",
      "    return predictions.sort(['prediction'],ascending=False)[:n]['movie_id'].values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "init = time()\n",
      "print getRecommendationsUser(data, 2, 10,SimPearson)\n",
      "print \"tiempo: \",time()-init\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[ 787 1420 1360  682 3233 3888 3607 3656  572 3245]\n",
        "tiempo:  23.67\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<b>Exercici 6:</b>\n",
      "\n",
      "Desenvolupa un sistema de recomanaci\u00f3 col\u00b7laboratiu basat en \u00edtems. \n",
      "\n",
      "Primer, escriu una funci\u00f3 <b>CalcSimItems(DataFrame)</b>, que construeixi i retorni una taula, itemsim, amb les semblances entre els \u00edtems.\n",
      "Despr\u00e9s escriu la funci\u00f3 principal, <b>getRecommendationsItem(DataFrame, itemsim, user, n)</b>, ha de tenir com a entrada les puntuacions dels usuaris, la taula de semblan\u00e7a entre \u00edtems, un \"user_id\" i el nombre n m\u00e0xim de recomanacions que volem. Com a sortida ha de donar les n millors pel\u00b7l\u00edcules."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "## Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimPearson(DataFrame,item2):\n",
      "    main_frame = DataFrame[DataFrame.movie_id == item2]\n",
      "    return coefPearson(main_frame['base_item_rating'].values, main_frame['rating'].values)\n",
      "\n",
      "\n",
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimEuclid(DataFrame,item2):\n",
      "    main_frame = DataFrame[DataFrame.movie_id == item2]\n",
      "    return 1 / ( 1 + distEuclid(main_frame['base_item_rating'].values, main_frame['rating'].values) )\n",
      "\n",
      "\n",
      "def CalcSimItems(DataFrame, similarity = SimEuclid):\n",
      "    #: listado con items unicos\n",
      "    unique_items = DataFrame['movie_id'].unique()\n",
      "    # partimos la lista en 2\n",
      "    #unique_items_a = unique_items[: int(len(unique_items)/2)]\n",
      "    #unique_items_b = unique_items[len(unique_items_a):]\n",
      "    #: montamos un dataFrame con ello y los asignamos como indices\n",
      "    itemsim = pd.DataFrame(index = unique_items)\n",
      "    itemsim.columns.name = 'movie_id'\n",
      "    # por cada item, calculamos su similitud respecto del resto\n",
      "    for item_b in unique_items:\n",
      "            \n",
      "        # generamos dataFrame con los usuarios que han votado ese item\n",
      "        base_item_frame = DataFrame[DataFrame.movie_id == item_b][['user_id','rating']]\n",
      "        # renombramos las columnas\n",
      "        base_item_frame.columns = ['user_id','base_item_rating']\n",
      "        # dataframe con los ratings ha otros items por los distintos users\n",
      "        base_no_item_frame = DataFrame[ DataFrame.movie_id != item_b][['user_id','movie_id', 'rating']]\n",
      "        # hacemos un dataframe a partir de un merge, donde mantenemos todos los usuarios que han votado el item principal\n",
      "        # y una columna con ese rating, ademas la id de los otros ratings a los otros items\n",
      "        base_frame = pd.merge(base_item_frame, base_no_item_frame, on='user_id')\n",
      "        \n",
      "        # calculamos las distancias\n",
      "        distancies = [ similarity(base_frame,item_a) for item_a in unique_items ]\n",
      "        itemsim[item_b] = distancies\n",
      "    # devolvemos itemsim    \n",
      "    return itemsim\n",
      "\n",
      "\n",
      "def getRecommendationsItem(data1, itemsim, usr, n,similarity=SimPearson):\n",
      "    \n",
      "    DataFrame = data1\n",
      "    user = usr\n",
      "    #: listado con items unicos\n",
      "    unique_items = DataFrame['movie_id'].unique()\n",
      "    #items del user\n",
      "    user_items =  DataFrame[DataFrame.user_id == user].movie_id.unique()\n",
      "    #seleccionamos items que no ha valorado el user\n",
      "    no_user_items = [ x for x in unique_items if x not in user_items ]\n",
      "    # creamos un dataframe de los items con rating\n",
      "    ratings = DataFrame[DataFrame.user_id == user][['movie_id','rating']]\n",
      "    ratings.index = ratings.movie_id.values\n",
      "    # creamos un listado solo de items\n",
      "    rated_items = DataFrame[DataFrame.user_id == user].movie_id.unique()\n",
      "    \n",
      "    results = []\n",
      "    #recorremos los items no valorados\n",
      "    for item_x in no_user_items:\n",
      "        simil = 0\n",
      "        x_simil = 0\n",
      "        # por cada item valorado\n",
      "        for item_y in rated_items:\n",
      "            simil += itemsim.ix[item_x][item_y]\n",
      "            x_simil += ( itemsim.ix[item_x][item_y] * ratings.ix[item_y]['rating'] )\n",
      "        \n",
      "        # normalizamos\n",
      "        results.append(x_simil / simil)\n",
      "        \n",
      "    # creamos dataframe de salida\n",
      "    predictions = pd.DataFrame(data = {'movie_id':  no_user_items, 'prediction':results })   \n",
      "    return predictions.sort(['prediction'],ascending=False)[:n]['movie_id'].values"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Executions\n",
      "#TARDA MUCHO TIEMPO CON EL DATAFRAME ENTERO: TESTEADO CON data[1000:75000]\n",
      "#data = data[1000:75000]\n",
      "itemsim=CalcSimItems(data)\n",
      "getRecommendationsItem(data, itemsim, 1, 10)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 24,
       "text": [
        "array([2628, 2268, 1357, 2916, 3068, 1210, 2194, 1537, 1213, 1103])"
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<b>Exercici 7:</b> Creu una funcio, <b>EvaluateRecommendationsUser(DataFrame,DataFrameTest,similarity=SimPearson)</b>, que donat un conjunt de dades d'entrenament i un conjunt de dades de test ens avalua la precisi\u00f3 dels sistema.\n",
      "Per a cadascun dels elements del conjunt de test haurem de pronosticar el seu valor i comparar-lo amb el valor real que l'usuari li ha asignat.<br> Els mesura que utilizarem per avaluar el sistema \u00e9s la seg\u00fcent:\n",
      "$$accuracy = 1/N\\sum_{i=0}^N  abs(rating_i - rating_i^*) $$ on rating \u00e9s la puntauci\u00f3 real que l'usuari va asginar a la pel\u00b7l\u00edcula i rating* \u00e9s el valor pronoticat pel sistema de recomanacio desenvolupat."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\"\"\" \n",
      "NECESITAMOS PARTES DEL 5, PERO LAS FUNCIONES HAN DE DEVOLVER TODOS LOS VALORES Y NO SOLO LOS MAS ALTOS\n",
      "\"\"\"\n",
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimPearson(DataFrame,User2):\n",
      "    main_frame = DataFrame[DataFrame.user_id == User2]\n",
      "    return coefPearson(main_frame['base_user_rating'].values, main_frame['rating'].values)\n",
      "\n",
      "\n",
      "# Returns a distance-based similarity score for person1 and person2 based on pearson distance\n",
      "def SimEuclid(DataFrame,User2):\n",
      "    main_frame = DataFrame[DataFrame.user_id == User2]\n",
      "    return 1 / ( 1 + distEuclid(main_frame['base_user_rating'].values, main_frame['rating'].values) )\n",
      "\n",
      "\n",
      "# Gets recommendations for a person by using a weighted average\n",
      "# of every other user's rankings\n",
      "def getRecommendationsUser(DataFrame,person,similarity=SimPearson):\n",
      "    # usuario base\n",
      "    base_user_id = person\n",
      "    # generamos dataframe con las movie_id del base user y sus ratings\n",
      "    base_user_frame = DataFrame[DataFrame.user_id == base_user_id][['movie_id','rating']]\n",
      "    # renombramos las columnas\n",
      "    base_user_frame.columns = ['movie_id', 'base_user_rating']\n",
      "    # generamos dataframe con los registros que esta el base user y solo las columnas movie_id user_id rating\n",
      "    base_no_user_frame = DataFrame[ DataFrame.user_id != base_user_id ][['movie_id', 'user_id', 'rating']]\n",
      "    # hacemos un dataframe a partir de un merge, donde mantenemos todas las movies del base_user (que son las que\n",
      "    # tienen en comun ambos dataframes), el rating del base user , y las id de los otros users con sus ratings\n",
      "    # : Por cada movie_id y rating del base user, estaran todos los demas\n",
      "    # : y tendremso los datos necesarios en el mismo registro\n",
      "    base_frame = pd.merge(base_user_frame,base_no_user_frame, on='movie_id')\n",
      "    # id's unicas de usuarios que no son base user\n",
      "    users_id = base_no_user_frame['user_id'].unique()\n",
      "\n",
      "    \n",
      "    #: dataframe de sortida\n",
      "    result_frame = pd.DataFrame()\n",
      "    #: distancias que acumularemos\n",
      "    distancies = [ similarity(base_frame,user_id) for user_id in users_id ]\n",
      "    # juntamos todos los calculos en un dataframe\n",
      "    similarities = pd.DataFrame(data = {'user_id':  users_id, 'similarity':distancies })\n",
      "    \n",
      "    # nos quedamos solo con los similarities mayores que 0\n",
      "    similarities = similarities[similarities.similarity > 0]\n",
      "    \n",
      "    # listado de peliculas unicas que hay en el dataframe\n",
      "    unique_movies = DataFrame['movie_id'].unique()\n",
      "    # creamos un listado de peliculas, unicas, y que no ha visto el user base\n",
      "    no_user_movies = [ x for x in unique_movies if x not in base_user_frame['movie_id'].unique() ]\n",
      "\n",
      "    # creamos un data_frame con las no_user_movies\n",
      "    recomendations = pd.DataFrame(data = no_user_movies, columns = ['movie_id'])\n",
      "    # hacemos un merge dataframe con : movie_id, user_id, rating : y no_user_movies : on movie_id\n",
      "    recomendations = pd.merge(recomendations, DataFrame[['user_id','movie_id','rating',]], on='movie_id')\n",
      "    # lo unimos con los similarities : on user_id\n",
      "    recomendations = pd.merge(recomendations,similarities, on='user_id')\n",
      "    \n",
      "    sim_sum_total = []\n",
      "    \n",
      "    no_user_movies = recomendations.movie_id.unique()\n",
      "    \n",
      "    #recorremos las movies no vistas y que pueden ser recomendadas\n",
      "    for movie in no_user_movies:\n",
      "        # extraemos los ratings y los similaritys de los user que la han valorado\n",
      "        step_data = recomendations[recomendations.movie_id == movie][['rating','similarity']]\n",
      "        # total rating \n",
      "        total = ( step_data.rating * step_data.similarity ).sum()\n",
      "        # total similarity\n",
      "        sim_sum = step_data.similarity.sum()\n",
      "        # prediccion\n",
      "        sim_sum_total.append( total / sim_sum )\n",
      "        \n",
      "    #lo metemos todo en un dataframe\n",
      "    predictions = pd.DataFrame(index = no_user_movies, data = {'prediction':sim_sum_total })\n",
      "    return predictions\n",
      "\n",
      "# evaluate the recomender given a training and a testing set. Return the accuracy of the system \n",
      "def EvaluateRecommendationsUser(DataFrame,DataFrameTest,similarity=SimPearson):\n",
      "    N = 0\n",
      "    total = 0\n",
      "    # recorremos los usuarios de test\n",
      "    for user in DataFrameTest.user_id.unique():\n",
      "        # extraemos los ratings reales del user\n",
      "        ratings = DataFrameTest[DataFrameTest.user_id == user][['movie_id', 'rating']]\n",
      "        # los indexamos\n",
      "        ratings.index = ratings.movie_id.values\n",
      "        # extraemos las predicciones\n",
      "        predictions = getRecommendationsUser(DataFrame, user, similarity)\n",
      "        # hacemos los calculos\n",
      "        for movie in ratings.movie_id.values:\n",
      "            total += abs(ratings.ix[movie]['rating'] - predictions.ix[movie]['prediction'])\n",
      "            N += 1\n",
      "            \n",
      "    return  total / N"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "accuracy = EvaluateRecommendationsUser(data[10:],data[0:9],similarity=SimEuclid)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print accuracy"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.622877546296\n"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}